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Overview 

This grant, Intelligent Command and Control Systems for Satellite Ground Operations, funded by NASA 
Goddard Space Flight Center, has spanned almost a decade. During this time, it has supported a broad 
range of research addressing the changing needs of NASA operations. It is important to note that many of 
NASA’s evolving needs, for example, use of automation to drastically reduce (e.g., 70%) operations costs, 
are similar requirements in both government and private sectors. 

Initially the research addressed the appropriate use of emerging and inexpensive computational 
technologies, such as X Windows, graphics, and color, together with COTS (commercial-off-the-shelf) 
hardware and software such as standard Unix workstations to re-engineer satellite operations centers. The 
first phase of research supported by this grant explored the development of principled design 
methodologies to make effective use of emerging and inexpensive technologies. The ultimate performance 
measures for new designs were whether or not they increased system effectiveness while decreasing costs. 

GT-MOCA (The Georgia Tech Mission Operations Cooperative Associate) and GT-VITA (Georgia 
Tech Visual and Inspectable Tutor and Assistant), whose latter stages were supported by this research, 
explored model-based design of collaborative operations teams and the design of intelligent tutoring 
systems, respectively. Implemented in proof-of-concept form for satellite operations, empirical evaluations 
of both, using satellite operators for the former and personnel involved in satellite control operations for the 
latter, demonstrated unequivocally the feasibility and effectiveness of the proposed modeling and design 
strategy underlying both research efforts. The proof-of-concept implementation of GT-MOCA showed that 
the methodology could specify software requirements that enabled a human-computer operations team to 
perform without any significant performance differences from the standard two-person satellite operations 
team. 

GT-VITA, using the same underlying methodology, the operator function model (OFM), and its 
computational implementation, OFMspert, successfully taught satellite control knowledge required by 
flight operations team members. The tutor structured knowledge in three ways: declarative knowledge 
(e.g., What is this? What does it do?), procedural knowledge, and operational skill. Operational skill is 
essential in real-time operations. It combines the two former knowledge types, assisting a student to use 
them effectively in a dynamic, multi-tasking, real-time operations environment. A high-fidelity simulator 
of the operator interface to the ground control system, including an almost full replication of both the 
human-computer interface and human interaction with the dynamic system, was used in the GT-MOCA 
and GT-VITA evaluations. The GT-VITA empirical evaluation, conducted with a range of ‘novices’ that 
included GSFC operations management, GSFC operations software developers, and new flight operations 
team members, demonstrated that GT-VITA effectively taught a wide range of knowledge in a succinct and 
engaging manner. 

As technology infusion became a standard part of GSFC satellite ground control operations, the 
research focus of this grant turned to address development and enhancement of methodologies to design 
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ground control operations that were increasingly efficient and cost effective. Methodologies sought to 
enhance operations performance with the result that the overall system cost was less and the number of 
anomalies that were detected and successfully resolved greater. This portion of the research included two 
projects: GT-IMaC and GT-FIXIT. GT-IMaC (Georgia Tech Interactive Monitoring and Control) proposed 
a methodology for the design of interfaces for real-time operations that concurrently enhance performance 
and engage operators in predominantly passive operations activities, such as extensive monitoring. The 
IMaC methodology combines OFM-based procedures to specify display function and form with human- 
computer interaction principles to design command-and-control interfaces that engaged operators 
performing a predominantly passive control task. Using NASA satellite controllers, the proof-of-concept 
demonstration and evaluation, using a high-fidelity satellite ground control simulator, showed how the 
resulting displays concurrently reduced the number of required operations personnel from two to one while 
significantly increasing performance and decreasing errors. 

GT-FIXIT (Georgia Tech Fault Information Extraction and Investigation Tool) addressed another 
aspect of operations: managing anomalies, many of which had been previously seen and resolved, that 
appeared new to operations personnel due to operator turnover that has quickly changed from twenty- five 
years to under twelve months. GT-FIXIT is a proof-of-concept demonstration of a methodology that 
combines model-based design and an artificial intelligence technique, case-based reasoning. The goal was 
to turn paper-based anomaly documentation, currently residing on clipboards in satellite mission support , 
as opposed to operations , rooms, into information for operators engaged in real-time control activities. The 
anomalies themselves were implemented as the lowest level cases in an electronic case base. Abstraction 
and aggregation provided layers to reduce potential redundancy and facilitate recognition of a potentially 
relevant anomaly class. Again a model of operations guided definition of case indices and the design of a 
user interface that helped operations personnel to identify quickly anomalies that occurred previously and 
resembled the current situation. Like GT-TIMI, using GSFC satellite controllers, GT-FIXIT demonstrated 
how the effective use of a case-based retrieval system with retrieval and user interface defined by a model- 
based methodology. The result, GT-FIXIT, applied to SAMPEX operations significantly enhanced fault 
management by concurrently improving the speed of fault detection and the accuracy of resolution. 

GT-FIXIT was a turning point both in the research funded by this grant and the direction of research 
that GSFC itself was pursuing. Due to drastic reductions in available funds for operations, a serious need 
arose to reduce operations costs by 70% after the initially funded life span of a spacecraft. Historically, 
though initial funding is for a short time, for example, the SAMPEX satellite was funded initially for 1 8 
months, NASA continued to fund a spacecraft as long as it is producing important science data. To date, 
almost ten years after launch, SAMPEX remains an active space vehicle for collecting science data. The 
new rules, however, mandate that operations, whose costs are primarily personnel, need to be greatly 
reduced. Automation is the widespread, and somewhat glib, answer to the question: “How are we going to 
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meet these goals?” A more challenging question is “How are we going to design and implement software 
systems to make such automation possible?” 

Almost serendipitously, GT-FIXIT provided one piece of necessary automation in a move to fully 
automated routine satellite control: a knowledge base of expertise based on passed experience that can be 
reused in similar situations. Reuse can occur because the anomaly case-based retrieval system helps less 
experienced personnel, as GT-FIXIT was initially intended to do, or because automation, running without 
human supervision, uses it to attempt to detect and resolve anomalies without loss of science data, 
canceling a pass, or requiring emergency human assistance. 

Following this research, this grant supported three related projects that specifically addressed issues 
associated with the design of operations automation. The first was an extensive field study of Genie — a 
GSFC prototype operations automation system designed to carry out satellite control in an unattended 
manner. The second was the design and proof-of-concept implementation of AutoPass, an OFMspert- 
based architecture intended to provide an alternative to Genie. The third is Apprentice. This system, 
building on AutoPass, specifies software that passively learns, incrementally refining its automation 
knowledge base by observing differences between predicted performance and that of human operators. 
Subsequently Apprentice provides tools to allow operations personnel to review differences and as 
necessary repair or enhance the automation knowledge base. 

The field study of Genie provided hard data, outside of aviation, of the strengths and weaknesses of a 
realistic system designed to automate operations that were previously carried out manually. Genie’s initial 
objective, similar to ‘lights-out’ automation of the 1980s in the automobile industry, was to reduce 
personnel by eliminating human-staffed shifts, the night shift being the first and managing satellites 
autonomously throughout weekends next. Genie was implemented and evaluated for the SAMPEX 
spacecraft in 1994. 

Georgia Tech researchers conducted an on-site field study to observe Genie’s behavior and the 
interaction of operations staff with Genie when there were problems or if a change to the normal Genie 
script was needed. This study included observation of 19 control sessions in which Genie could potentially 
be used. During this study, the first of a two-part field study, many problems were observed. If the metric 
was the extent to which Genie met the goal of lights-out automation, none of the observed passes was 
successful. Each required real-time human intervention. 

The results of the first study were so surprising, and NASA found them so shocking, that Georgia 
Tech agreed to conduct the same study a second time, after SAMPEX operators had more than an 
additional year of experience with Genie. Again Georgia Tech researchers conducted on-site observation 
of SAMPEX operations and observed 18 control sessions in which Genie could potentially be used. The 
results of the second study were remarkably similar to those of the first. One can only conclude that the 
structure and operation of Genie itself made independent operation unreliable and brittle. 
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It is important, however, to acknowledge Genie’s significant contribution to automated satellite 
operations. Genie was the firs/ such system designed and expected to perform satellite control 
autonomously. Without a baseline system, subsequent Georgia Tech research would have been impossible; 
both follow-on research efforts exploited Genie’s strengths and carefully developed principles to mitigate 
observed weaknesses. Any initial system would manifest unexpected flaws. All following systems would 
use both observed flaws and strengths as a baseline. A baseline is prerequisite to evolving automation and 
the evolution of design principles for software systems that more closely meet automation needs. Genie 
provided an excellent baseline and included many significant features included in subsequent systems. 

AutoPass is a pass automation system that builds on OFMspert and the OFM. It was designed to 
mimic Genie functionality for the SAMPEX spacecraft. Because it is derived from the OFM model and the 
OFMspert architecture, AutoPass inherits capabilities such as heterarchy, nondeterminism, and a robust 
operations model that represents a range of operations with which any certified operator would be expected 
to cope — both nominal and off nominal. As a result AutoPass successfully manages a much wider range of 
passes than Genie, without operations personnel in attendance. In addition, an empirical evaluation 
compared the performance of SAMPEX flight operations personnel in detecting, diagnosing, and 
suggesting software-level changes to mitigate similar failures in the future. Data show that AutoPass 
allowed operators to understand why AutoPass failed, that is to specify the conditions in the software that 
caused the problem, and to suggest software- level changes to modify AutoPass in order to eliminate same 
problem recurring in the future. In almost all cases, Genie did not allow operators these insights into its 
operations. Overall, AutoPass demonstrated that it possessed a number of desired characteristics of 
automation. The most important criticisms found in many increasingly automated commercial aircraft 
include the finding that automation is opaque, strong but brittle, and silent. AutoPass operations were 
visible, not opaque. Operations staff could typically understand what it was doing and why. AutoPass was 
robust , not brittle. Since AutoPass extends the OFMspert architecture, which various other experiments 
have shown matches operations activity more than 90% of the time, AutoPass responds appropriately to a 
wider range of environmental conditions and command-and-control requirements. Finally, AutoPass is 
inspectable , mitigating problems of strong, silent, and error-prone automation. Operators could not only 
understand what AutoPass software was doing and why, but they could also make software- level 
suggestions to improve the system. 

The next obvious question is “Can automation software be designed to allow it to acquire operations 
knowledge without direct human input, for example, via interviews with domain subject matter experts, 
and, once operational, can operations personnel, as opposed to software engineers, repair it when it fails?’ 
Apprentice, also based on OFMspert and building on both AutoPass and its associated evaluation, seeks to 
address these issues. As an OFMspert derivative, Apprentice begins with a fairly robust model of 
operations. Lights-out automation, however, requires a broader and more flexible knowledge base than 
systems such as operator aids or associates. Thus, Apprentice increases its knowledge base by learning . It 
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runs in real time concurrently with operations personnel who initially test the system during pre-launch and 
subsequently ensue that it is stable after launch and during early orbit. Watching in the background, 
Apprentice compares its responses to operator responses and store scenarios where differences are detected. 
Post real-time operations, Apprentice provides capabilities that allow operators to view scenarios in which 
differences occurred and provides tools that enable operators to modify the Apprentice knowledge base, 
both domain knowledge and operational procedures. Apprentice, as this grant concludes, is approaching 
the point where it can tested to evaluate the extent to which the hypotheses that underpin Apprentice are 
valid, creating both a new type of learning system and specifying an automation architecture that operators, 
not software engineers, can repair. A proof-of-concept implementation and evaluation will take place for 
the TRACE spacecraft. 

GT-MOCA: The Georgia Tech Mission Operations Cooperative Associate 

One approach to aiding the human supervisory controller of a complex dynamic system is to provide an 
intelligent operator's associate. The GT-MOCA research proposes a prescriptive theory of human- 
computer cooperative problem solving and describes the design and evaluation of a prototype system based 
on the theory. The theory consists of five principles: human-in-charge, mutual intelligibility, openness and 
honesty, management of trouble, and multiple perspectives. 

A prototype intelligent associate system, the Georgia Tech Mission Operations Cooperative Assistant 
(GT-MOCA), is an embodiment of these principles that provides a collection of context-sensitive resources 
for the human operator of a simulated satellite ground control system. These resources include an 
interactive visualization of current activities, organized message lists of important events, and interactive 
graphics depicting the current state of the controlled system. An evaluation utilizing NASA satellite 
ground controllers showed that GT-MOCA was perceived to be useful and provided performance benefits 
for certain portions of the control task. 

This research resulted in numerous publications, including Dr. Patricia M. Jones’ doctoral thesis from 
the Georgia Institute of Technology. The most significant include the following: 

Jones, P. M. (1991). Human-computer cooperative problem solving in supervisory control. Unpublished 
Ph.D. thesis, Center for Human-Machine Systems Research, School of Industrial and Systems 
Engineering, Georgia Institute of Technology, Atlanta, GA. 

Jones, P. M., & Mitchell, C. M. (1994). Model-based communicative acts: Human-computer collaboration 
in supervisory control. International Journal of Human-Computer Studies, 41, 527-551. 

Jones, P. M., Chu, R. W., & Mitchell, C. M. (1995). A methodology for human-machine systems research: 
Knowledge engineering, modeling, and simulation. IEEE Transactions on Systems, Man, and 
Cybernetics, 25(7), 1025-1038. 

Jones, P. M., & Mitchell, C. M. (1995). Human-computer cooperative problem solving: Theory, design, 
and evaluation of an intelligent associate system. IEEE Transactions on Systems, Man, and 
Cybernetics , 25(7), 1039-1053. 
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The most focused description of the GT-MOCA work appeared in IEEE Transactions on Systems, Man, 
and Cybernetics , 1995. This paper is included in this report as Appendix A. 

GT-VITA: Georgia Tech Visual and Inspectable Tutor and Assistant 

Training is a critical issue for operators responsible for the safe and efficient operation of large-scale complex 
dynamic systems. This project proposes and articulates a set of requirements for an intelligent tutoring system. The 
requirements specify what (instructional content) and how (instructional strategies) to teach a novice operator to 
supervise and control a complex dynamic system. The instructional content teaches system structure and behavior 
(i.e., declarative knowledge), system procedures (i.e., procedural knowledge), and how to use this declarative and 
procedural knowledge to manage a complex dynamic system in real time (i.e., operational skill). Using the 
underlying representations of the operator function model (OFM) and OFMspert, the OFM's computational 
implementation, GT-VITA (Georgia Tech Visual and Inspectable Tutor and Assistant) realizes these requirements. 
As a proof-of-concept demonstration, an instance of the generic GT-VITA tutoring architecture was implemented 
for satellite ground controllers. The empirical evaluation, utilizing NASA satellite ground control personnel, 
showed that GT-VITA was a flexible and effective training system. In fact, NASA adopted VITA as the foundation 
for required training for all satellite ground control personnel and uses a variation to this day. 

In addition to an intelligent tutoring system architecture, by using and extending the operator function model 
and OFMspert, GT-VITA demonstrates a robust methodology for conceptualizing the tutor-aid paradigm. The 
tutor-aid paradigm defines a conceptual framework in which learning with an intelligent tutor gradually becomes 
collaboration with an intelligent associate. Using the same structures (i.e., OFM and OFMspert) and the same 
domain knowledge, GT-VITA specifies a tutor and GT-MOCA (Jones & Mitchell, 1995) specifies an aid. 

The GT-VITA research resulted in numerous publications, including Dr. Rose M. Chu’s doctoral 
thesis from the Georgia Institute of Technology. The most significant include the following: 

Chu, R. W. (1991). Towards the tutor/aid paradigm: Design of intelligent tutoring systems for operators of 
supervisory > control systems. Ph.D. dissertation, Center for Human-Machine Systems Research, School 
of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA, Atlanta, GA. 

Chu, R. W., Mitchell, C. M., & Jones, P. M. (1995). Using the operator function model and OFMspert as 
the basis for an intelligent tutoring system: Towards a tutor/aid paradigm for operators of supervisory 
control systems. IEEE Transactions on System , Man , and Cybernetics, 25(7), 1054-1075. 

The most succinct, yet complete, description of the GT-VITA work appeared in IEEE Transactions on 
Systems, Man, and Cybernetics , 1995. This paper is included in Appendix B of this report. 

GT-IMaC (Georgia Tech Interactive Monitoring and Control) 

The Georgia Tech Interactive Monitoring and Control (GT-IMaC) methodology is a design methodology 
for operator interfaces to complex engineering systems. This four-step methodology helps designers tailor 
human-computer interaction to communicate monitoring and control activity requirements to the operator. 


flnal.rpt.8.99 


7 



Printed: August 25, 1 999 


saved: 8/25/99 9:28 AM 


The goal is an interface that engages the operator in the task of monitoring to ensure that fault detection 
occurs in an efficient and reliable way. Figure 1 summarizes the principle steps of the IMaC methodology. 


operator system 

interviews domain engineer 

arxi review observation interviews 


i i _ i 



Figure 1. Principle steps of the IMaC design methodology 


To determine the effectiveness of the proposed methodology a proof-of-concept operator interface 
was designed for NASA satellite controllers. An experiment was conducted at NASA’s Goddard Space 
Flight Center. Eight satellite operators served as subjects. After training sessions on both a replication of 
the conventional interface and the IMaC interface, subjects monitored and controlled a simulated satellite 
ground control system for eight scenarios. In keeping with standard activities in the domain, each scenario 
was approximately ten minutes in length and contained eight subsystem faults, a control activity, and a 
number of record keeping activities. Dependent measures included fault detection times and rates, 
procedure completion times, and procedure execution error rates. The resulting data show that use of the 
interface designed using the IMaC methodology was a significant determinant of operator performance. 
Sessions in which subjects used the IMaC interface consistently showed better performance. Fault detection 
times were significantly higher and fault detection rates were lower. In addition, subjects made 
significantly fewer errors while executing control procedures. At the conclusion of the experimental 
evaluation, subjects were asked to rate the overall usefulness of the IMaC interface and its individual 
components. All subjects had a very favorable opinion of the interactive interface and preferred it to the 
conventional interface currently in use at NASA. As a result, NASA has incorporated many of the IMaC 
interface features used in the proof-of-concept demonstration in the standard set of interface widgets 
available to new satellite teams as they specify the operational control interface for new systems. 
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The GT-IMaC research resulted in numerous publications, including. David A. Thurman’s M.S. 
thesis from the Georgia Institute of Technology. The most significant publications include the following: 

Thurman, D. A., & Mitchell, C. M. (1994). A methodology for the design of interactive monitoring 

interfaces. Proceedings of the 1994 IEEE International Conference on Systems, Man , and Cybernetics , 
San Antonio, TX, 1738-1744. 

Thurman, D. A., & Mitchell, C. M. (1995). A design methodology for operator displays of highly 

automated supervisory control systems. Proceedings of the 6th IFAC/IFIP/IFOR/SEA Symposium on 
Analysis, Design , and Evaluation of Man-Machine Systems , Cambridge, MA, 821-825. 

Thurman, D. A. (1995). Improving operator effectiveness in monitoring complex systems : A methodology / 
for the design of interactive monitoring and control interfaces . M.S. Thesis, Center for Human- 
Machine Systems Research, School of Industrial and Systems Engineering, Georgia Institute of 
Technology, Atlanta, GA. 

Thurman, D. A. (1997). The interactive monitoring and control (IMaC) design methodology: Application 
and empirical results. Proceedings of the Human Factors and Ergonomics Society ? 4 1st Annual 
Meeting , Albuquerque, NM, 289-293. 

The two best descriptions of the GT-IMaC research are provided in Appendix C. 

GT-FIXIT: The Georgia Tech Fault Information Extraction and Investigation Tool 

The GT-FIXIT research explores the use of case-based reasoning as a technique for storing fault 
management experience and making it available to operators confronting similar anomalous situations. 
Specifically, this project investigates the use of case-based reasoning technology to construct a knowledge 
base of actual fault management experience. This knowledge base is organized so as to enable the retrieval 
of this fault management information in response to system inputs. 

GT-FIXIT was implemented in proof-of-concept form for SAMPEX satellite controllers. Due to 
very high operator turnover, and the current convention of storing anomaly documentation in a separate 
room, on paper, and organized by date of occurrence, many previously seen anomalies appear new to less 
experienced operations staff. GT- FIXIT proposed turning archival data into information useful for fault 
management in real-time operations. In addition to storing previously anomalies as cases in a case base, 
GT-FIXIT also used a model-based design methodology to specify case indices and design the interactive 
user interface that helped operators identify similar anomalies and associated fault management techniques 
in real time. Figure 2 depicts the GT-FIXIT architecture. 

To evaluate the effectiveness of GT- FIXIT in supporting real-time fault management, an experiment 
was conducted using satellite ground control operators at NASA Goddard Space Flight Center. Eight 
subject teams, a pair of operators, one a certified NASA satellite ground controller and the other an 
experimental confederate, controlled a simulated satellite ground control system during eight satellite 
control scenarios. Scenarios were matched by fault type and each team saw four fault categories under each 
of the two conditions: GT-FIXIT and conventional control room tools including those for fault 
management. 
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Figure 2. The GT-FIXIT architecture 


Six performance measures were used to assess subjects’ ability to detect, identify, and formulate an 
appropriate solution to anomalies. Use of GT-FIXIT resulted in significantly improved subject performance 
for all measures. Subjects detected anomalies, identified them as replicas or variants of previous 
anomalies, and determined the correct response significantly more quickly with GT-FIXIT than when using 
conventional operations interface and available fault management tools. Subjects’ anomaly identifications 
and responses were also evaluated for correctness: accuracy of diagnosis and response were also 
significantly higher with GT-FIXIT. 

The GT-FIXIT research resulted in numerous publications, including. Andrew J. Weiner’s masters 
thesis (in progress) from the Georgia Institute of Technology. The most significant publications include the 
following: 

Weiner, A. J., & Mitchell, C. M. (1995). FIXIT: A case-based architecture for computationally encoding 
fault management experience. Proceedings of the 6th IFAC/IFIP/IFOR/SEA Symposium on Analysis, 
Design, and Evaluation of Man Machine Systems , Boston, MA, 196—200. 

Weiner, A. J., Thurman, D. A., 8c Mitchell, C. M. (1995). Applying case-based reasoning to aid fault 
management in supervisory control. Proceedings of the 1995 International Conference on Systems, 
Man, and Cybernetics , Vancouver, BC, 4213-4223. 

Weiner, A. J., Thurman, D. A., 8c Mitchell, C. M. (1996). FIXIT: An architecture to support recognition- 
primed decision making in complex system fault management. Proceedings of the Human Factors and 
Ergonomics Society, 40th Annual Meeting , Philadelphia, PA, 209-213. 

Weiner, A. J. (in progress). FIXIT: A case-based architecture for computationally encoding fault 
management experience. , Center for Human-Machine Systems Research, M.S. thesis, School of 
Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA. 

Two comprehensive publications of the GT-FIXIT research are provided in Appendix D. 
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Field Study of Genie 

Operations automation is a computer-based system that attempts to encode operator expertise, typically the 
expertise of personnel in charge of complex dynamic systems. The use of automation to replace operators 
is becoming a serious option in many systems, some of which are safety critical. 

One application of operations automation is ‘lights-out’ automation, that is, systems in which there are 
no operators to ‘tend’ the system. If an error occurs, the system remotely pages an operator who evaluates 
the problem. The field study described in this research is predicated on the assumption that even the best- 
designed automation requires human interaction, since, for the foreseeable future, all automation will 
inevitably fail at some point in time. 

Automated systems that attempt to emulate human expertise are likely to fail even more often, as 
building robust systems of this type is a new and challenging design issue. Thus, identifying and 
addressing design issues for operations automation are critical components to ensure safe and productive 
systems. Design issues include the ability to elicit and encode robust expert knowledge, as well as 
supporting human-automation interaction with operations personnel when the system reaches its inevitable 
limits, and with engineers and software designers who must ensure that the system does not fail the same 
way more than once. This project describes a field study of operations automation fielded at NASA for 
satellite control. The purpose of this study is to understand how and why such automation fails and identify 
design issues that facilitate or degrade human-automation interaction. The field study of Genie had two 
parts: the first conducted in February 1996 and the second in June 1997. Because Genie, like many 
advanced technology systems, was evolving as problems were encountered, NASA felt a second study was 
needed to ensure that the data from the first remained representative and accurate. The goal of these studies 
was to observe how Genie performed and identify problems operators encountered while interacting with 
Genie. The same researcher was present during all passes for both studies. The researcher took notes to 
document potential problems or anomalies in real time during each pass, while event logs from the 
controlled system and Genie activities were archived to files. Post-pass, the researcher asked operations 
personnel present during the pass to explain observ ed problems or anomalies, and to comment on 
individual Genie activities. Figure 3 depicts the combined data for these studies. 

Combining both studies, a total of thirty-seven passes were observed. In seven, operators chose not to 
use Genie due to one or more perceived limitations. In the remaining passes, Genie was used eleven times 
in advisory mode and nineteen times in the fully automatic mode. A categorization scheme to organize the 
data from passes using Genie was proposed. The expectation was that categories would highlight potential 
design limitations. Figure 3 uses the categorization as the x-axis. Statistical analysis shows no significant 
differences between the problems observed in the two field studies. 

Based on the results of these studies and the analysis above, a set of design features was formulated. 
They are intended to facilitate future designs of similar systems. Feature 1. Operations automation must be 
robust (category 1 — automation-control system links; category 3 — automation capabilities; and various 
reasons for canceling the use of Genie in advance). Feature 2. The knowledge base and reasoning system 
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must be inspectable and easily modified (category 2 — automation anomalies). Feature 3. Operations 
automation must facilitate correction of erroneous operator input (category 5 — recovery from operator 
error). Feature 4. Operations automation should facilitate testing (category 7 — task coordination). Feature 
5. Standard software engineering practices must be observed to ensure that operations automation will 
operate successfully within a ‘lights-out’ environment (category 4 — automation flexibility; category 6 — 
environmental 
limitations). 



Figure 3. Summary data for the two Genie field studies 

Field studies such as these are necessary and important. They provide real-world data that designers 
and human factors practitioners can use to improve the effectiveness of advanced technology systems. As 
with most systems, the design cycle must include several iterations of testing, evaluation, and refinement. 
Particularly because the very concept of operations automation, which by definition is somewhat rigid — a 
computer program does exactly what the programmer tells it to do — and knowledge acquisition and 
representation remain both challenging and open research issues, it is important to study any fielded system 
to identify prototypical human interaction problems and enhancements to advance the design of future 
systems. 

The Genie field study resulted in numerous publications, contributing to David M. Brann ’s M.S. 
thesis (in progress) and David A Thurman’s Ph.D. thesis (in progress), both from the Georgia Institute of 
Technology. The most significant publications include the following: 

Brann, D. M., Thurman, D. A., & Mitchell, C. M. (1995). Case-based reasoning as a methodology for 

accumulating human expertise for discrete system control. Proceedings of the 1995 IEEE International 
Conference on Systems, Man, and Cybernetics , Vancouver, BC, 4219-4223. 

Brann, D. M., Thurman, D. A., & Mitchell, C. M. (1996). Human interaction with lights-out automation: A 
field study. Proceedings of the 1996 Symposium on Human Interaction in Complex Systems, Dayton, 
OH, 276-283. 

Thurman, D. A., Brann, D. M., & Mitchell, C. M. (1997). An architecture to support incremental 

automation of complex systems. Proceedings of the 1997 IEEE International Conference on Systems, 
Man , and Cybernetics , Orlando, FL, 1 174-1179. 
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Thurman, D. A., Brann, D. M., & Mitchell, C. M. (1998). The human factor in ’lights-out’ automation: 

Using field study data to identify critical human factors design issues. Proceedings of the Human 
Factors and Ergonomics Society 42nd Annual Meeting , San Diego, 364-368. 

Brann, D. M. (in progress). Design of robust operations automation for safety-critical system: Field Studies 
and an automation architecture , AutoPass. M.S. thesis, Center for Human-Machine Systems Research, 
School of Industrial & Systems Engineering, Georgia Institute of Technology, Atlanta, Ga. 

Thurman, D. A. (in progress). Apprentice : An architecture for operations automation that passive learns 
via operator observation and allows operators to repair and extend its knowledge base . Ph.D. thesis, 
Center for Human-Machine Systems Research, School of Industrial & Systems Engineering, Georgia 
Institute of Technology, Atlanta, Ga. 

Appendix E contains the publication with the most comprehensive description of this research. 

AutoPass: An OFMspert-Based Automation for ‘Lights-Out’ Satellite Operations 

AutoPass was designed and implemented to mitigate some of the problems both observed during the field 
studies and predicted by the cognitive engineering community. AutoPass possesses the same control 
functionality as Genie, but is based on a very different underlying architecture. It extends the operator 
function model (OFM) and OFMspert methodology. The research hypothesis is that operations automation 
designed by extending this architecture will be more robust and, when failure inevitably occurs, allow 
operators, as opposed to software engineers, to understand why the automation failed and suggest methods 
to repair observed problems. 

AutoPass uses OFMspert as a foundation and extends it to meet the needs of satellite ground control. 
The intelligence in AutoPass primarily resides in OFMspert ’s operations model, a computer 
implementation of the OFM. The OFM, developed and validated over almost two decades, represents 
operator activities as a hierarchic-heterarchic of finite-state nodes. Nodes correspond to operator activities 
and decompose down to physical, cognitive, and perceptual operator actions. The heterarchic and non- 
deterministic properties of the OFM respectively give the model the ability to represent the multi-task 
nature of many control situations and permit a range of activity choices at low levels. Operations are rarely 
as sequential as models or even user manuals suggest, and operators routinely exploit this flexibility. 

Models of human behavior, or systems such as Genie, typically do not represent such flexibility. This 
contributes to a great deal of the brittleness found in many systems. 

OFMspert contains a current state space with state variables and derived variables. The latter 
semantically represents state information that operators sample and aggregate to assess the system or 
progress of activity sequences such as procedures. For example, AutoPass has state variables that specify 
how many commands are sent to the spacecraft and how many commands the spacecraft actually receives. 
One of derived variables is commandCountlnvalid , which is true when these counters do not match. When 
true, the derived variable triggers a troubleshooting activity that operators commonly use to determine why 
the counters do not have the same value. Troubleshooting heuristics and procedures often used by 
operators, encoded in the OFM, attempt to accomplish a task in spite of an initial failure. 
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This example provides a context to illustrate a number of AutoPass features that attempt to mitigate 
operations automation problems. Derived variables and inclusion of standard troubleshooting activities are 
human attributes of control. They model human flexibility, heuristics, and workarounds. Built on the 
OFM/OFMspert methodology, AutoPass possesses these features automatically and thus potentially 
reduces the brittleness found in similar systems. 

Eight certified SAMPEX controllers participated in an AutoPass evaluation. Each operator observed 
four automatically controlled passes under two different satellite control architectures: Genie and AutoPass. 
Participants were told that they were reviewing the results of lights-out control, a replay of passes carried 
out previously, possibly over the past weekend. At various points in each pass, both control systems 
encountered nonstandard events. At each point, the experimenter paused the system and asked participants 
to describe the current situation, why it was occurring, identify whether or not the automation was 
functioning correctly, and, when relevant, suggest how to fix the problem. 

Initial data from the evaluation show that with AutoPass, controllers were much more likely to be able 
to explain what the failure or degradation was, why it had occurred from the level of software operation, 
and suggest software-level changes to eliminate the problem in the future. Overall, AutoPass appears to 
incorporate many of the design principles of human-centered automation including visibility and robust 
operation. 

The AutoPass project resulted in numerous publications, contributing to David M. Brann’s M.S. 
thesis (in progress) and David A Thurman’s Ph.D. thesis (in progress), both from the Georgia Institute of 
Technology. The most significant publications include the following: 

Thurman, D. A., Brann, D. M., & Mitchell, C. M. (1998). The human factor in 'lights-out' automation: 

Using field study data to identify critical human factors design issues. Proceedings of the Human 
Factors and Ergonomics Society 42nd Annual Meeting, San Diego, 364-368. 

Thurman, D. A., Brann, D. M., & Mitchell, C. M. (1999). Operations automation: Definition, examples, 
and a human-centered approach. Proceedings of the Human Factors and Ergonomics Society 43rd 
Annual Meeting, Houston, to appear. 

Brann, D. M. (in progress). Design of robust operations automation for safety-critical system: Field Studies 
and an automation architecture , AutoPass. M.S. thesis, Center for Human-Machine Systems Research, 
School of Industrial & Systems Engineering, Georgia Institute of Technology, Atlanta, Ga. 

Thurman, D. A. (in progress). Apprentice: An architecture for operations automation that passive learns 
via operator observation and allows operators to repair and extend its knowledge base. Ph.D. thesis, 
Center for Human-Machine Systems Research, School of Industrial & Systems Engineering, Georgia 
Institute of Technology, Atlanta, Ga. 

Appendix F contains the most descriptive publication of this research. 

Apprentice 

AutoPass was designed as an initial response to a specific system — Genie. Given the success of AutoPass 
for a specific application a more general architecture was warranted. Apprentice is a more rigorous and 
principled extension of OFMspert. It posits a theory of operations automation that meets many of the 
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obvious problems and addresses a pervasive problem endemic to most knowledge-based systems (KBS) — 
knowledge elicitation. 

Obtaining necessary knowledge is a fundamental problem with most automated systems, particularly 
those that depend on obtaining and encoding operator knowledge. Such knowledge is often tacit and highly 
contextualized. Current knowledge elicitation techniques fail to address this problem adequately. As a 
result, knowledge engineering is often identified as the bottleneck in construction of effective knowledge- 
based systems, including operations automation. Since a knowledge base that is as extensive and flexible as 
the human operators it is replacing is a prerequisite for effective operations automation, design of such 
systems is entirely dependent on more effective knowledge engineering techniques. In particular, 
alternatives to traditional knowledge engineering techniques, typically involving interviewing subject 
matter experts, are sorely needed. 

Apprentice is an alternative to traditional methods. Exploiting the proven power of the OFM and 
OFMspert methodology, Apprentice is an extension of OFMspert designed to monitor expert operators 
controlling a system and detect and record inconsistencies. As with any OFMspert applications, given its 
OFM-based operations model, OFMspert posits expectations about what, when, and how an operator will 
carry out control activities. Apprentice extends OFMspert by encapsulating situations in which expert 
operators and OFMspert fail to agree, and, retrospectively, provides operators with the tools to replay the 
situation, determine why the difference occurred, and in many situations to actually repair the OFM-based 
operations model. 

Apprentice is based on the assumption that successful operations automation must be understandable, 
repairable, and extensible by human operators who currently carry out system control functions. Apprentice 
defines a human-centered architecture in which operations knowledge is represented in a manner that is 
understandable by operations personnel. In the initial phases, Apprentice performs the activity its name 
implies: it acts as an apprentice to expert operators. It watches operators control the system while 
concurrently developing its own hypotheses and noting where mismatches occur. Post hoc , Apprentice 
provides tools to enable operations personnel to reconstruct and inspect points of mismatch between 
Apprentice predictions and actual operator activities. Mismatches are typically model errors or limitations. 
Finally, Apprentice provides tools that enable operators, rather than software engineers, to repair or extend 
the knowledge base underpinning the evolving operations automation system. 

Using the operator function model, Apprentice represents knowledge in ways similar to those operators 
use to organize knowledge about how to carry out required activities: hierarchical, heterarchical collections 
of activities to meet operational goals given system constraints. Moreover, the OFM inherently allows 
flexibility at both high and low levels, allowing operators to encode a variety of styles or techniques by 
which to carry out control. The OFM/OFMspert methodology provides graphical representations of 
activities and constraints, methods used to carry out those activities, and information needed to initiate, 
enable, complete, or terminate them. Previous research, including the AutoPass research, shows that the 
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OFM/OFMspert methodology provides a powerful knowledge representation which operators can easily 
understand and validate. 

Apprentice is both a theory and a software architecture. It embodies a three-phase process for 
developing an operations automation knowledge base. In phase one, initial knowledge base definition , an 
OFM-based operations model for OFMspert is constructed. Although it uses some traditional knowledge 
engineering techniques, OFM construction also incorporates extensive on-site and iterative validation and 
model refinement. Domain experts use sophisticated graphical and interactive representations of both the 
OFM and OFMspert to inspect the knowledge base and validate the runtime expectations. Again, the 
purpose is to ensure model predictions and actual operations match. The initial Apprentice knowledge 
based is obtained when this process concludes. 

In the incremental knowledge elicitation and verification phase , Apprentice ‘observes’ human 
operators who carry out control activities and compares them with predictions based on the initial 
OFMspert operations model. Apprentice identifies discrepancies between predicted activities and actual 
operator activities. These discrepancies provide ‘contextualized’ learning opportunities for Apprentice. 
Operators repair and extend the operations model (i.e., knowledge base) using specialized editors. 

Once the operations model has been sufficiently verified and extended to manage a range of situations 
typically encountered in the domain, Apprentice transitions its ' lights-out ’ operations phase. It executes 
activities based on the constraints and conditions defined in the OFMspert operations model. Apprentice, 
based on an OFM representation of operator activities, carries out activities in ways that are similar to those 
of human operators. Thus, human operators can understand and repair discrepancies when they arise. 

There are three primary advantages to the Apprentice approach to operations automation. First, its 
human-centered knowledge representation technique and associated graphical editors enables domain 
practitioners to inspect, repair and extend the knowledge base. Second, Apprentice elicits operational 
knowledge in the context of actual operations, resulting in a more robust knowledge base than obtained 
with traditional methods. Third, Apprentice’s ability to observe compare actual and predicted operator 
activities and its support for knowledge base editing by domain practitioners means those practitioners 
should only have to handle a new situation one time. When new problems are encountered, an on-call 
operator resolves the problem and then updates the Apprentice knowledge base to address the situation in 
the future. The next time the situation is encountered, Apprentice has the requisite knowledge to address it. 

Apprentice is being implemented as a proof-of-concept operations automation prototype for NASA’s 
TRACE (Transition Region and Coronal Explorer) satellite. An empirical evaluation will be conducted to 
address phases two and three of the Apprentice approach: the extent to which operators can understand and 
repair discrepancies and the extent to which the result successfully and automatically controls routine 
TRACE passes. Pass types and associated anomalies used in the assessment are those with which a newly 
certified TRACE operator would be expected to easily cope. 

Apprentice makes several important contributions to the design of operations automation. First, 
its use of the OFM/OFMspert methodology ensures that the knowledge base has a human-centered 
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representation with associated graphical editors that enable domain practitioners to inspect, repair, and 

extend it. Second, Apprentice adds a powerful new strategy to the repertoire of knowledge engineering 

techniques. It blends the two competing strategies: machine learning and direct knowledge elicitation. 

Apprentice begins with a robust model of operations. It then exploits the power of both the computer and 

OFMspert to validate and modify the initial knowledge base by passively identifying discrepancies, likely 

model errors or limitations. Next, it creates an environment and provides tools such that domain 

practitioners, post hoc , can view scenarios containing discrepancies and resolve them. Third, Apprentice 

supports operations automation that evolves. Tools allow domain practitioners to detect, understand, and 

repair discrepancies so that the operations automation learns — it will never fail the same way twice. The 

same tools support extensions to the knowledge base as the dynamic system changes. 

The Apprentice project, near to completion, has already resulted in numerous publications. Moreover 
it is the major contribution of David A Thurman’s Ph.D. thesis (in progress) from the Georgia Institute of 
Technology. The most significant publications to date include the following: 

Thurman, D. A., Brann, D. M., & Mitchell, C. M. (1997). An architecture to support incremental 

automation of complex systems. Proceedings of the 1997 IEEE International Conference on Systems, 
Man, and Cybernetics , Orlando, FL, 1 174-1 179. 

Thurman, D. A., Brann, D. M., & Mitchell, C. M. (1998). The human factor in ’lights-ouf automation: 
Using field study data to identify critical human factors design issues. Proceedings of the Human 
Factors and Ergonomics Society 42nd Annual Meeting , San Diego, 364-368. 

Thurman, D. A., & Mitchell, C. M. (1998). Apprentice: A technique for building operations automation. 
Proceedings of the 1998 IEEE International Conference on Systems, Man, and Cybernetics , Chicago, 
IL, 967-972. 

Thurman, D. A., Brann, D. M., & Mitchell, C. M. (1999). Operations automation: Definition, examples, 
and a human-centered approach. Proceedings of the Human Factors and Ergonomics Society 43rd 
Annual Meeting, Houston, to appear. 

Thurman, D. A. (in progress). Apprentice: An architecture for operations automation that passive learns 
via operator observation and allows operators to repair and extend its knowledge base. Ph.D. thesis, 
Center for Human-Machine Systems Research, School of Industrial & Systems Engineering, Georgia 
Institute of Technology, Atlanta, Ga. 

Appendix G contains the most descriptive publication of this research. 


OFM and OFMspert Evolution 

Underlying threads for all of these research projects are two complementary modeling and design 
methodologies: the operator function model (OFM) and OFMspert, the computational implementation of 
the OFM. With various applications, OFMspert has acquired increased in functionality and has become more stable 
and both domain and platform independent. The current version of OFMspert reads domain-specific information from 
files. OFMspert design, that is, specification of the required files to tailor the system to a specific domain and 
application, is undertaken concurrently with the development of the graphical representation of the OFM. Major files 
are those that define OFM activities and the contents of the Current Problem Space. 
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In addition to the domain and platform independence, the OFM and OFMspert have merged from two separate 
but related entities into one. The range of tools provided to implement or correct OFMspert applications also provides 
tools that to concurrently construct an OFM. This OFM is 100% consistent with OFMspert code and domain- 
dependent files that define events, variables, operator activities, etc. Historically, the OFM was conceptual, implement 
with computer-based drawing tools such as Claris Draw and Visio. As a conceptual model, the transition from an 
existing OFM to an OFMspert application was often not straightforward. It easy to use a conceptual model with 
people, but it makes a poor software specification. Merging the OFM/OFMspert models allows the construction of the 
same conceptual representation as the former implementations of the OFM did, but it has constrained the vocabulary, 
activity definitions, and variables, to exactly those that OFMspert needs for a particular application. This is a 
significant step forward for the OFM/OFMspert methodology. 

The most recent description of the evolving OFM/OFMspert methodology is given in the paper 
below. It is also included in Appendix H. 

Thurman, D. A., Chappell, A. R., & Mitchell, C. M. (1998). An enhanced architecture for OFMspert: A 
domain-independent system for intent inferencing. Proceedings of the 1998 IEEE International 
Conference on System , Man and Cybernetics , San Diego, CA, 955-960. 

Miscellaneous Publications Presenting High-Level, Integrated Views of this Research 

In addition to papers describing individual projects as they evolved and were completed, this grant 
supported the preparation of several chapters in recent books and papers, summarizing significant research 
programs and contribution. One is the introduction to a special issue of the IEEE Transactions on Systems, 
Man, and Cybernetics . Another is a chapter summarizing and integrating of all the GSFC-related research 
conducted prior to 1990, most conducted prior to this grant. Finally, a recent chapter proposes an 
engineering methodology to enable the use of the OFM and OFMspert, using many of the projects 
sponsored by this grant as example. 

Mitchell, C. M. (1996). GT-MSOCC: Operator models, model-based displays, and intelligent aiding. In W. 
B. Rouse (Ed.), Human/technology interaction in complex systems (Vol. 8, pp. 67-172). Greenwich, 
CT: JAI Press Inc. 

Mitchell, C. M., & Sundstrom, G. A. (1997). Human interaction with complex systems: Design issues and 
research approaches. IEEE Transactions on Systems, Man, and Cybernetics, 27(3), 265-273. 

Mitchell, C. M. (1999). Model-based design of human interaction with complex systems. In A. P. Sage & 
W. B. Rouse (Eds.), Systems engineering & management handbook (pp. 745-810). New York, NY: 
John Wiley. 

All three of these publications are provided in Appendix I. 
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Appendix A 


GT-MOCA: 

The Georgia Tech Mission Operations Cooperative Associate 
Related Publications* 

Jones, P. M. (1991). Human-computer cooperative problem solving in supervisory control. Unpublished 
Ph.D. thesis, Center for Human-Machine Systems Research, School of Industrial and Systems 
Engineering, Georgia Institute of Technology, Atlanta, GA. 

Jones, P. M., & Mitchell, C. M. (1994). Model-based communicative acts: Human-computer collaboration 
in supervisory control. International Journal of Human-Computer Studies, 41, 527-551. 

Jones, P. M., Chu, R. W., & Mitchell, C. M. (1995). A methodology for human-machine systems research: 
Knowledge engineering, modeling, and simulation. IEEE Transactions on Systems, Man, and 
Cybernetics, 25(7), 1025-1038. 

Jones, P. M., & Mitchell, C. M. (1995). Human-computer cooperative problem solving: Theory, design, 
and evaluation of an intelligent associate system. IEEE Transactions on Systems, Man, and 
Cybernetics, 25(7), 1039-1053. 


* Copies of these publications can be obtained from www.chmsr.gatech.edu/publications/ or by contacting 
Dr. Christine M. Mitchell. 
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Appendix B 


GT-VITA: 

Georgia Tech Visual and Inspectable Tutor and Assistant 


Related Publications* 

Chu, R. W. (1991). Towards the tutor/aid paradigm: Design of intelligent tutoring systems for operators of 
supervisory’ control systems. Ph.D. dissertation, Center for Human-Machine Systems Research, School 
of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA, Atlanta, GA. 

Jones, P. M., Chu, R. W., & Mitchell, C. M. (1995). A methodology for human-machine systems research: 
Knowledge engineering, modeling, and simulation. IEEE Transactions on Systems, Man, and 
Cybernetics, 25(7), 1025-1038. 

Chu, R. W., Mitchell, C. M., & Jones, P. M. (1995). Using the operator function model and OFMspert as 
the basis for an intelligent tutoring system: Towards a tutor/aid paradigm for operators of supervisory 
control systems. IEEE Transactions on System, Man, and Cybernetics, 25(7), 1054-1075. 


* Copies of these publications can be obtained from www.chmsr.gatech.edu/publications/ or by contacting 
Dr. Christine M. Mitchell. 


final. rpt.8.99 


20 



Printed: August 25, 1999 


saved: 8/25/99 9:28 AM 


Appendix C 


GT-IMaC: 

Georgia Tech Visual and Inspectable Tutor and Assistant 
Related Publications* 


Thurman, D. A., & Mitchell, C. M. (1994). A methodology for the design of interactive monitoring 

interfaces. Proceedings of the 1994 IEEE International Conference on Systems, Man, and Cybernetics , 
San Antonio, TX, 1738-1744. 

Thurman, D. A., & Mitchell, C. M. (1995). A design methodology for operator displays of highly 

automated supervisory control systems. Proceedings of the 6th I FAC/1 FI P/I FOR/SEA Symposium on 
Analysis, Design, and Evaluation of Man-Machine Systems, Cambridge, MA, 821-825. 

Thurman, D. A. (1995). Improving operator effectiveness in monitoring complex systems: A methodology 1 
for the design of interactive monitoring and control interfaces. M.S. Thesis, Center for Human- 
Machine Systems Research, School of Industrial and Systems Engineering, Georgia Institute of 
Technology, Atlanta, GA. 

Thurman, D. A. (1997). The interactive monitoring and control (IMaC) design methodology: Application 
and empirical results. Proceedings of the Human Factors and Ergonomics Society 4 1st Annual 
Meeting , Albuquerque, NM, 289-293. 


* Copies of these publications can be obtained from www.chmsr.gatech.edu/publications/ or by contacting 
Dr. Christine M. Mitchell. 
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Appendix D 


GT-FIXIT: 

Georgia Tech Visual and Inspectable Tutor and Assistant 
Related Publications* 

Weiner, A. J., & Mitchell, C. M. (1995). FIXIT: A case-based architecture for computationally encoding 
fault management experience. Proceedings of the 6th IFAC/IFIP/IFOR/SEA Symposium on Analysis, 
Design , and Evaluation of Man Machine Systems, Boston, MA, 196—200. 

Weiner, A. J., Thurman, D. A., & Mitchell, C. M. (1995). Applying case-based reasoning to aid fault 
management in supervisory control. Proceedings of the 1995 International Conference on Systems, 
Man, and Cybernetics , Vancouver, BC, 4213-4223. 

Weiner, A. J., Thurman, D. A., & Mitchell, C. M. (1996). FIXIT: An architecture to support recognition- 
primed decision making in complex system fault management. Proceedings of the Human Factors and 
Ergonomics Society , 40th Annual Meeting , Philadelphia, PA, 209-213. 

Weiner, A. J. (in progress). FIXIT: A case-based architecture for computationally encoding fault 
management experience . , Center for Human-Machine Systems Research, M.S. thesis, School of 
Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA. 


* Copies of these publications can be obtained from www.chmsr.gatech.edu/publications/ or by contacting 
Dr. Christine M. Mitchell. 
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Appendix E 


Genie Field Studies: 


Related Publications* 


Brann, D. M., Thurman, D. A., & Mitchell, C. M. (1995). Case-based reasoning as a methodology for 

accumulating human expertise for discrete system control. Proceedings of the 1995 IEEE International 
Conference on Systems, Man, and Cybernetics , Vancouver, BC, 4219-4223. 

Brann, D. M., Thurman, D. A., & Mitchell, C. M. (1996). Human interaction with lights-out automation: A 
field study. Proceedings of the 1996 Symposium on Human Interaction in Complex Systems , Dayton, 
OH, 276-283. 

Thurman, D. A., Brann, D. M., & Mitchell, C. M. (1997). An architecture to support incremental 

automation of complex systems. Proceedings of the 1997 IEEE International Conference on Systems, 
Man , and Cybernetics , Orlando, FL, 1 174-1 179. 

Thurman, D. A., Brann, D. M., & Mitchell, C. M. (1998). The human factor in 'lights-ouf automation: 
Using field study data to identify critical human factors design issues. Proceedings of the Human 
Factors and Ergonomics Society 42nd Annual Meeting , San Diego, 364-368. 

Brann, D. M. (in progress). Design of robust operations automation for safety -critical system: Field Studies 
and an automation architecture , AutoPass. M.S. thesis, Center for Human-Machine Systems Research, 
School of Industrial & Systems Engineering, Georgia Institute of Technology, Atlanta, Ga. 

Thurman, D. A. (in progress). Apprentice : An architecture for operations automation that passive learns 
via operator observation and allows operators to repair and extend its knowledge base. Ph.D. thesis, 
Center for Human-Machine Systems Research, School of Industrial & Systems Engineering, Georgia 
Institute of Technology, Atlanta, Ga. 


* Copies of these publications can be obtained from www . chmsr . gatech . edu/publ ic ations/ or by contacting 
Dr. Christine M. Mitchell. 
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Appendix F 


AutoPass: 

OFMspert-Based Automation for Satellite Operations 


Related Publications* 


Thurman, D. A., Brann, D. M., & Mitchell, C. M. (1998). The human factor in ’lights-out’ automation: 

Using field study data to identify critical human factors design issues. Proceedings of the Human 
Factors and Ergonomics Society 42nd Annual Meeting , San Diego, 364-368. 

Thurman, D. A., Brann, D. M., & Mitchell, C. M. (1999). Operations automation: Definition, examples, 
and a human-centered approach. Proceedings of the Human Factors and Ergonomics Society 43rd 
Annual Meeting , Houston, to appear. 

Brann, D. M. (in progress). Design of robust operations automation for safety-critical system: Field Studies 
and an automation architecture, AutoPass. M.S. thesis, Center for Human-Machine Systems Research, 
School of Industrial & Systems Engineering, Georgia Institute of Technology, Atlanta, Ga. 

Thurman, D. A. (in progress). Apprentice: An architecture for operations automation that passive learns 
via operator observation and allows operators to repair and extend its knowledge base. Ph.D. thesis, 
Center for Human-Machine Systems Research, School of Industrial & Systems Engineering, Georgia 
Institute of Technology, Atlanta, Ga. 


* Copies of these publications can be obtained from www.chmsr.gatech.edu/publications/ or by contacting 
Dr. Christine M. Mitchell. 
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Appendix G 


Apprentice: 

An OFMspert-Based Operations Automation Architecture that Learns and Allows 

Repair by Operations Personnel 

Related Publications* 

Thurman, D. A., Brann, D. M., & Mitchell, C. M. (1997). An architecture to support incremental 

automation of complex systems. Proceedings of the 1997 IEEE International Conference on Systems, 
Man, and Cybernetics , Orlando, FL, 1174-1179. 

Thurman, D. A., Brann, D. M., & Mitchell, C. M. (1998). The human factor in 'lights-out' automation: 
Using Field study data to identify critical human factors design issues. Proceedings of the Human 
Factors and Ergonomics Society > 42nd Annual Meeting , San Diego, 364-368. 

Thurman, D. A., & Mitchell, C. M. (1998). Apprentice: A technique for building operations automation. 
Proceedings of the 1998 IEEE International Conference on Systems, Man, and Cybernetics , Chicago, 
IL, 967-972. 

Thurman, D. A., Brann, D. M., & Mitchell, C. M. (1999). Operations automation: Definition, examples, 
and a human-centered approach. Proceedings of the Human Factors and Ergonomics Society 43rd 
Annual Meeting , Houston, to appear. 

Thurman, D. A. (in progress). Apprentice: An architecture for operations automation that passive learns 
via operator observation and allows operators to repair and extend its knowledge base . Ph.D. thesis, 
Center for Human-Machine Systems Research, School of Industrial & Systems Engineering, Georgia 
Institute of Technology, Atlanta, Ga. 


* Copies of these publications can be obtained from www.chmsr.gatech.edu/publications/ or by contacting 
Dr. Christine M. Mitchell. 
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Appendix H 


The Evolving Operator Function Model and OFMspert: 


Related Publications* 

Thurman, D. A., Chappell, A. R., & Mitchell, C. M. (1998). An enhanced architecture for OFMspert: A 
domain-independent system for intent inferencing. Proceedings of the 1998 IEEE International 
Conference on System, Man and Cybernetics , San Diego, CA, 955-960. 


* Copies of these publications can be obtained from www.chmsr.gatech.edu/publications/ or by contacting 
Dr. Christine M. Mitchell. 


final.rpt.8.99 


26 



Printed: August 25, 1999 


saved: 8/25/99 9:28 AM 


Appendix I 


Miscellaneous Publications: 


High-Level Integrative Summaries of Various Related Research Projects 


Mitchell, C. M. (1996). GT-MSOCC: Operator models, model-based displays, and intelligent aiding. In W. 
B. Rouse (Ed.), Human/technology interaction in complex systems (Vol. 8, pp. 67-172). Greenwich, 
CT: JAI Press Inc. 

Mitchell, C. M., & Sundstrom, G. A. (1997). Human interaction with complex systems: Design issues and 
research approaches. IEEE Transactions on Systems, Man, and Cybernetics, 27(3), 265-273. 


Mitchell, C. M. (1999). Model-based design of human interaction with complex systems. In A. P. Sage & 
W. B. Rouse (Eds.), Systems engineering & management handbook (pp. 745-810). New York, NY: 
John Wiley. 


* Copies of these publications can be obtained from www.chmsr.gatech.edu/publications/ or by contacting 
Dr. Christine M. Mitchell. 
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